Chapter 1 of 9 · 2017

The original Transformer, one component at a time

I wanted to understand how we got from the 2017 Transformer to DeepSeek V4.1 Flash. The short answer is that nobody threw the original away. Each new model swapped out a few parts and kept the rest. That makes the original worth knowing well, so this first chapter is only about the 2017 model. I go box by box through the diagram, follow one token through an encoder layer, and then spend most of the page on attention, because that is the part everything later keeps changing.

Source: Vaswani et al., Attention Is All You Need (2017)Reading time ≈ 14 min3 interactive figures

The series at a glance

Where we are. The filled dot is this chapter. Later chapters reuse the same diagram and the same colours, and only highlight the parts that changed.

01 / Components

The whole machine in one picture

The 2017 paper was about translation, so the model has two halves. The encoder reads the whole source sentence and turns each word into a vector. The decoder writes the translation one word at a time, and at every step it can look back at what the encoder produced. Hover over or tap a block to see what it does and when it gets replaced later in the series.

Figure 1 · Encoder–decoder Transformer, redrawn from the paper's Figure 1

Pick a block

Six identical layers on the left, six on the right, and data flows upward. The little loops around each block are residual connections: the block's output is added to its input rather than replacing it.

Pick a component to see what it does and when it gets replaced.

Embeddings & positionsAttentionAdd & LayerNormFeed-forwardOutput head

What I left out. Dropout, the √d_model scaling on the embeddings, and the fact that the two embedding tables and the final Linear layer share their weights. The blue path is worth a second look. It is the only connection between the two halves, and every one of the six decoder layers reads from it.

6 + 6encoder layers + decoder layers

512d_model, width of the residual stream

8 × 64attention heads × head width

2048hidden width of the feed-forward block

65Mparameters, base model

213Mparameters, big model (d_model 1024, 16 heads)

~37kshared byte-pair vocabulary, WMT14 En–De

12 hbase training, 8 P100 GPUs, 100k steps

02 / One encoder layer

Follow one token through one encoder layer

Now zoom in on one encoder layer. A token comes in as a list of 512 numbers and leaves as a different list of 512 numbers. Two things happen to it on the way. First, attention lets it pull in information from the other tokens in the sentence. Then a small feed-forward network reworks it on its own, without looking at any other token. In both cases the result is added to what was already there rather than swapped in, and then normalised.

Step 1 of 6

A token arrives as a vector

Each of the n tokens in the sentence is a row of 512 numbers: its embedding plus its positional encoding. The whole sentence is an n × 512 matrix, and the layer processes all rows at once.

Post-LayerNorm order, as in the 2017 paper: LayerNorm(x + Sublayer(x)). Chapter 2 moves the norm in front of each sub-layer, which is what almost every model since GPT-2 does.

Attention is the only cross-token step

Every other block in the layer treats each token independently. Attention is where the word cat can pull in information from sat. If you remove attention, the model becomes a per-token lookup table.

The feed-forward block holds most of the parameters

Two matrices, 512 × 2048 and 2048 × 512, give about 2.1M parameters per layer. The four attention projections (Q, K, V, output) are 512 × 512 each, about 1.05M per layer. That two-to-one ratio is why later chapters replace the feed-forward block with a Mixture of Experts.

03 / Encoder self-attention

How self-attention actually works

This is the mechanism the paper is named after. Each token produces three vectors from its own embedding. The query is roughly what the token is looking for, the key is what it has to offer, and the value is the information it hands over if picked. To update a token, you compare its query against every key, turn those scores into weights that add up to one, and take the weighted average of the values.

Attention(Q, K, V) = softmax( Q Kᵀ / √d_k ) Vd_k = 64 in the base model. Without the division, dot products grow with d_k, the softmax turns into a hard pick of one token, and gradients vanish. Dividing by √64 = 8 keeps it soft.

Figure 3 · Scaled dot-product attention, one head, drawn as matrix shapes

Follow the shapes. The score matrix is n by n, one row and one column per token, so doubling the sentence length quadruples the work. That is the cost later chapters keep trying to bring down. Notice too that nothing in this picture knows what order the words came in. That has to be added to the inputs beforehand, which is section 05.

Figure 4 · Attention weights for one query, interactive illustrative values

Encoder attention looks both ways. Pick a word and you can see it attending to words before it and after it. The arcs and the highlighted row of the matrix are the same six numbers drawn two ways. I made the numbers up to look plausible. They are not from a trained model.

04 / Multi-head

Why eight heads instead of one

With a single set of attention weights per layer, the model would have just one notion of which words matter to which. The paper runs eight of them side by side instead. Each head works on its own 64-dimensional slice of the 512-dimensional token, computes its own weights, and produces its own output. One head might end up tracking which adjective belongs to which noun while another tracks what a pronoun refers to. The eight outputs are stitched back together into 512 numbers and passed through one more matrix.

Figure 5 · Multi-head attention: split, attend in parallel, concatenate, project

Why this matters later. When a model is generating text, it keeps the keys and values of every previous token around so it does not have to recompute them. With eight heads per layer, that is eight sets of K and V per token per layer. This is the KV cache, and it gets very large for long inputs. Four of the later chapters are mostly about making it smaller: grouped-query attention (chapter 3), multi-head latent attention (chapter 5), compressed sparse attention (chapter 8), and the 890 bytes per token of V4.1 Flash (chapter 9).

05 / Positions

Telling the model where each word is

If you shuffled the words in a sentence, attention would produce the same outputs, just shuffled the same way. It has no idea about order. The paper's fix is to add a position-dependent vector to every token before the first layer. The vector is built from sines and cosines at 256 different frequencies, so each position gets its own pattern.

PE(pos, 2i) = sin( pos / 10000^2i/512 ) PE(pos, 2i+1) = cos( pos / 10000^2i/512 )The first dimensions change quickly from one position to the next, the last ones very slowly. A useful side effect: moving k positions forward always looks like the same rotation, whatever the starting position, so the model can learn relative distances.

Figure 6 · The sinusoidal positional encoding, first 64 of 512 dimensions, positions 0 to 49 computed live

Each row is one position's vector. Blue is +1, copper is −1. Read down a column and you see one sine or cosine wave, fast on the left and slower to the right. I am only showing 64 of the 512 columns. The rest are so slow they barely change across a normal sentence. Chapter 3 swaps this for rotary embeddings, which put the same idea inside the attention computation instead of adding it to the input.

06 / Encoder vs decoder

Encoder attention versus decoder attention

In the encoder, every word can look at every other word. The whole source sentence is there from the start, so there is nothing to hide. The decoder is different. It is being trained to predict the next word, so it must not be allowed to see it. The fix is a mask: before the softmax, every score for a word to the right of the current one is set to minus infinity, so its weight comes out as zero. That mask is the only difference between the two kinds of self-attention in Figure 1.

Figure 7 · Which keys each query may attend to

Chapter 2 keeps only the right-hand grid. GPT-2 throws away the encoder and the cross-attention and uses the masked pattern everywhere. So does everything after it. Even V4.1 Flash, which brings back something it calls an encoder, masks it the same way. The grid on the left does not come back in this series.

Encoder self-attention

Every word sees every other word. It runs once over the whole input, and its output is what cross-attention reads.

This is the right tool when you have the whole input up front, as in translation, classification, or computing embeddings.

Decoder self-attention

Each word sees only itself and the words before it. When generating, every new token attends to the ones already written, and their keys and values are kept in a cache instead of being recomputed.

That cache is why long contexts are expensive, and most of the attention changes from chapter 3 onward are attempts to shrink it.

07 / The ledger

What the next eight chapters change

Each row is one part of Figure 1, and the last column says which chapter changes it. This table is also how I picked the models for the series: a model gets a chapter if it changed something here.

Component	2017 Transformer	What replaces it	Where
Overall shape	Encoder + decoder, cross-attention between them	Decoder-only stack	Ch 2 · GPT-2
Normalisation	Post-LayerNorm	Pre-LayerNorm, then RMSNorm	Ch 2 Ch 3 · LLaMA
Positions	Fixed sinusoids added to embeddings	Learned absolute positions, then rotary (RoPE) applied to Q and K	Ch 2 Ch 3
Feed-forward	512 → 2048 → 512 with ReLU	Gated SwiGLU, then a Mixture of Experts with a router	Ch 3 Ch 4 · Mixtral Ch 5 · DeepSeek-V2
Attention heads	8 heads, each with its own K and V	Grouped-query attention shares K, V across heads; multi-head latent attention compresses K, V into one small latent per token	Ch 3 Ch 5
Which tokens attend	Every token to every token	An indexer selects a sparse top-k of keys; then keys and values are compressed 4:1 and 128:1 and mixed with a 128-token sliding window	Ch 7 · V3.2 Ch 8 · V4
Routing & balance	None, dense compute	Shared + fine-grained experts, auxiliary-loss-free balancing, node-limited routing	Ch 5 Ch 6 · V3
Prediction head	One next-token softmax	A multi-token prediction module, later a separate speculative drafter	Ch 6 Ch 9
Residual stream	One 512-wide stream, plain addition	Four parallel streams mixed by a learned doubly-stochastic matrix (manifold-constrained hyper-connections)	Ch 8 · V4
Encoder / decoder split	Bidirectional encoder, causal decoder, cross-attention	Gone from chapter 2, then back as a causal encoder whose output feeds the decoder's global KV	Ch 2 Ch 9 · V4.1 Flash

Two things never got replaced. Blocks still add their output to a running residual stream rather than overwriting it, and one token still reads another through a softmax over query and key scores. V4.1 Flash widens the stream to four lanes and compresses most of its keys, but it keeps both ideas.

08 / Sources and method

Sources and method

Vaswani et al., "Attention Is All You Need", NeurIPS 2017 (arXiv:1706.03762)
All numbers in the fact strip: Sections 3.1 to 3.5 (architecture), Table 3 (base and big configurations, 65M and 213M parameters), Section 5 (training: 8 P100 GPUs, 100k steps in 12 hours, ~37k BPE vocabulary for En–De). Figure 1 here is redrawn from the paper's Figure 1.
Rush et al., "The Annotated Transformer"
Used to check the post-LayerNorm ordering and the parameter-count arithmetic for the attention and feed-forward blocks.
Peter Gostev, "Original Transformer vs DeepSeek"
The comparison that made me want to write this series. It goes straight from 2017 to 2026, and I wanted to see the steps in between.

I drew the figures as SVG by hand from the paper's equations and its Figure 1. Where a figure shows numbers a trained model would produce, like the attention weights in Figure 4, I made them up and say so. The positional encoding heat map and the score grids are computed in the page from the formulas shown.