In June 2017, eight researchers at Google posted a paper about machine translation. It proposed a network with no recurrence and no convolutions, trained it for three and a half days on eight GPUs, and beat every published translation system on the standard English to German benchmark. The title was a joke on a Beatles song: Attention Is All You Need.
Nine years later, nearly everything people call “AI” descends from the diagram in that paper. ChatGPT, Claude, Gemini, LLaMA and the DeepSeek models all use the same basic machine with parts swapped out. I wanted to understand exactly which parts, so I am writing a visual series that goes through the changes one model at a time. This post is the short version of the history, and an introduction to that series.
The problem the paper solved
Before 2017, the best translation systems were recurrent networks. They read a sentence one word at a time, carrying a hidden state forward, and wrote the translation the same way. Attention already existed as a helper for these systems: Bahdanau, Cho and Bengio had shown in 2014 that letting the decoder look back at every source word improved translations.
The bottleneck was the recurrence itself. Each step depended on the previous one, so a long sentence could not be processed in parallel. Training was slow, and long-range dependencies degraded as they passed through many steps.
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin removed the recurrence and kept only the attention. Every token looks at every other token in one parallel step. Order comes from a fixed positional signal added to the inputs. The result was faster to train, easier to scale, and better.
That is really why it took over. A model that parallelises well can be made bigger, and over the next few years bigger turned out to be the thing that mattered.
What happened next
The paper’s diagram had two halves, an encoder that reads and a decoder that writes. In 2018 each half became a product line. OpenAI’s GPT-1 kept the decoder and trained it to predict the next word. Google’s BERT kept the encoder and trained it to fill in blanks. Both showed that pretraining on raw text, then adapting to a task, beat training from scratch.
After that, the main thing that changed was size. GPT-2 in 2019 had 1.5 billion parameters. GPT-3 in 2020 had 175 billion and could perform tasks from a few examples in the prompt, with no fine-tuning at all. The same year, a paper from OpenAI showed that loss falls predictably as compute, data and parameters grow, which gave labs a reason to keep spending. Transformers also crossed into vision with ViT and into protein structure prediction with AlphaFold 2.
Then, on November 30, 2022, OpenAI put a chat interface in front of a fine-tuned GPT-3.5 and called it ChatGPT. It reached a hundred million users in about two months. That was when most people first noticed, but the architecture underneath had barely changed in five years.
The most recent stretch has been about cost and open weights. Meta released LLaMA’s weights in 2023, and the open ecosystem adopted its recipe of RMSNorm, SwiGLU and rotary positions. Mixtral showed that a mixture of experts could match a much larger dense model. DeepSeek, a lab in Hangzhou, then spent 2024 and 2025 attacking the two costs that scale worst, the key-value cache and the number of tokens that attention has to touch. Its R1 model in January 2025 matched frontier reasoning performance with open weights and a published training cost that briefly erased around six hundred billion dollars of Nvidia’s market value in a day. Its V4.1 Flash, released this month, runs a million-token context with a cache of under a kilobyte per token.
What has not changed
Put the 2017 diagram next to any of these models and you can still match up the parts. Two ideas in particular have never been replaced. The first is the residual stream, where each block adds its result onto a running vector rather than replacing it. The second is softmax attention over query and key scores, the mechanism by which one token reads another.
Everything else has been swapped at least once. Layer normalisation moved before the block, then became RMSNorm. Sinusoidal positions became learned positions, then rotary ones. The feed-forward block gained a gate, then became a router in front of hundreds of experts. Attention heads learned to share keys and values, then to compress them into a small latent, then to skip most tokens entirely. V4.1 Flash even brings back a form of the encoder and decoder split, although both halves are now causal.
I wanted a way to see those changes rather than read about them. Peter Gostev built a striking three-dimensional comparison of the 2017 model against V4.1 Flash, but it jumps straight from the first frame to the last. The series I am publishing fills in the steps between.
The series
Each chapter takes one architecture, redraws the same block diagram, and highlights only the components that changed. Every chapter has hand-drawn SVG figures, at least one interactive figure, and a fact strip with numbers taken from the primary paper or model card. The planned chapters:
- The original Transformer (2017)
- GPT-2 and the decoder-only stack (2019)
- LLaMA: RMSNorm, SwiGLU, RoPE and grouped-query attention (2023)
- Mixtral and the sparse mixture of experts (2023)
- DeepSeek-V2: multi-head latent attention (2024)
- DeepSeek-V3: load balancing and multi-token prediction (2024)
- DeepSeek-V3.2: sparse attention with a lightning indexer (2025)
- DeepSeek-V4: compressed attention and hyper-connections (2026)
- DeepSeek V4.1 Flash: the causal encoder-decoder (2026)
Chapter 1 is up. It goes through the original model’s parts, follows one token through an encoder layer, and works through how self-attention computes its weights, with queries, keys, values and the eight heads. It ends with a table of what each later chapter replaces.
Open chapter 1: the original Transformer →
Opens in a new tab. The page is self-contained, responsive, and uses no external JavaScript or web fonts.
Later chapters will be linked here as they are published.
Sources
- Vaswani et al., Attention Is All You Need, 2017. Training details in Section 5, results in Table 2.
- Bahdanau, Cho and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, 2014.
- Radford et al., Improving Language Understanding by Generative Pre-Training, 2018. Devlin et al., BERT, 2018.
- Brown et al., Language Models are Few-Shot Learners, 2020. Kaplan et al., Scaling Laws for Neural Language Models, 2020.
- DeepSeek-AI, DeepSeek-V4.1-Flash model card and release note, September 2026.