01 — The one idea
Everything is communication, then computation
In 2017, Attention Is All You Need replaced recurrence with a deceptively simple mechanism: let every token decide, at runtime, which other tokens matter to it.
Before: one word at a time
An RNN reads left to right, folding each new word into a running hidden state. Word 400 must wait for words 1–399, and the early ones get squeezed through 400 multiplications. Long-range dependencies fade; training can't parallelize over the sequence.
After: everything at once
Attention compares every token with every other token directly — one hop, regardless of distance. No sequential dependency, so the whole sequence processes in parallel on a GPU. That single change is why models could suddenly get enormous.
The whole architecture in one sentence: alternate two operations — attention, where tokens exchange information, and a feed-forward network, where each token thinks about what it just received — and repeat that block 12 to 100+ times.
One thing to hold onto: the architecture has barely changed since 2017. Almost every "new" model is the same block with better normalization, better position encoding, and better attention kernels. Learn the block and you've learned all of them.
02 — The pipeline
Follow one sentence through the model
From raw characters to a probability for the next token. Click any stage.
Notice there is no "understanding" step and no grammar module. Everything above is learned matrix multiplication, and the only structural decisions are how wide, how deep, and how many tokens can look at each other.
03 — Tokens & positions
The model never sees letters
Text is chopped into subword units from a fixed vocabulary, each mapped to an integer. Type anything below.
Why subwords?
A character vocabulary makes sequences painfully long; a whole-word vocabulary can't handle typos, names, or "unbelievably". Subword tokenizers (BPE, SentencePiece) land in between: common words stay one token, rare ones split into reusable pieces. unbelievable → un + believ + able.
Those pieces are shared, so the model learns morphology for free: -ing, -ed, un- behave consistently wherever they appear. Typical vocabularies run 32k–200k entries.
Why positions?
Attention is permutation-invariant: it computes dot products, and dot products don't care about order. Feed it "dog bites man" and "man bites dog" shuffled and you get the same multiset of outputs. Position information must be injected explicitly.
Old way: add a sinusoidal or learned vector to each embedding. Modern way: RoPE, which rotates each query and key by an angle proportional to its position — so the attention score between two tokens depends on their relative offset, which generalizes better to lengths never seen in training.
04 — Self-attention
The only place tokens talk
Each token asks a question (query), every token advertises what it has (key), and the ones that answer well contribute their contents (value).
One learned projection of the token's vector. It is compared against every key.
Another projection, one per token. The query·key dot product measures relevance.
A third projection holding the actual content. Values are averaged using the attention weights.
Why divide by √dk? Dot products of random vectors grow with dimension: at d=64, scores land around ±8 instead of ±1. Softmax of large numbers is nearly one-hot, so almost all gradient vanishes and learning stalls. Scaling keeps scores in a sane range. It's a tiny detail that decides whether training works.
Who is each word looking at?
What attention is not
It isn't a lookup table of grammar rules, and the weights aren't "explanations". They're a soft, learned routing: token A copies a weighted blend of token B's value, and the next layer does something useful with that blend. Meaning emerges across layers, not in any single head.
The party analogy
Every token walks into a room and asks "who here knows about what I need?" It scores everyone's one-line self-description, spends its attention budget on the best matches, and comes away with a summary of what they said. Then it thinks privately (feed-forward), and the next room has a new question.
05 — Multi-head attention
Many conversations at once
One attention operation can only express one notion of relevance per token. So run several in parallel at lower dimension, then concatenate. Same cost, far more expressive.
Same sentence, three heads. Each panel is a full attention matrix — rows are queries, columns are keys, brightness is weight. Hover a cell.
Real models have 16–128 heads, and nobody assigns them jobs. Training discovers specializations on its own: previous-token heads, induction heads that copy patterns, heads that track quotation and syntax, and "attention sink" heads that dump weight on the first token as a no-op. A 32-head layer is 32 partly-overlapping views of the same sequence, and the output projection learns how to combine them.
Why split instead of one big head? One head of width 4096 has one softmax distribution — it must commit to a single relevance pattern. Thirty-two heads of width 128 each get their own distribution, and together they can track syntax and coreference and position simultaneously. The total parameter count and FLOPs are identical.
06 — The rest of the block
Attention is only half of it
Three more ingredients turn the idea into a trainable deep stack: normalization, residual connections, and a private thinking step.
Residual connections
Every sublayer output is added back to its input: y = x + f(x). Gradients flow straight down that addition, so a 96-layer network trains about as easily as a 6-layer one. This single line, borrowed from ResNet, is what made depth practical.
Normalization
LayerNorm (or RMSNorm) re-scales each token's vector so activations don't drift as depth grows. RMSNorm drops mean-centering — cheaper, equally good — and is what most current models use.
The feed-forward network
After attention mixes information across positions, the FFN transforms each position independently: expand to ~4× width, apply a nonlinearity, project back. It holds roughly two-thirds of all parameters.
Intuition: attention finds the relevant context; the FFN computes with it. Mechanistic interpretability suggests the first matrix detects features, the activation selects which are present, and the second writes associated facts — closer to a key-value memory than to a classic MLP.
07 — Three shapes
Same block, different masks
The mask is the only real architectural fork. It decides which tokens are allowed to see which other tokens — and therefore what the model can be trained to do.
Sees everything
Bidirectional attention. Trained by hiding ~15% of tokens and predicting them. Produces rich contextual representations — excellent for classification, retrieval, and embeddings. Can't generate fluently.
when to reach for it
Any task where you need a good vector for a span of text: search, reranking, NER, sentiment, clustering. Smaller, cheaper, and often more accurate than asking a chat model.
Sees only the past
A causal mask blocks every future position. The payoff: one forward pass yields a next-token prediction at every position, so one sequence gives N training signals. This is why it won.
why generation works
Because each position was trained to predict only from its own past, the model can be run on a prefix and asked for one new token — then that token is appended and the process repeats. Same function, applied incrementally.
Reads, then writes
An encoder reads the input bidirectionally; a decoder generates causally and periodically attends into the encoder's output through cross-attention.
when it still wins
When input and output are genuinely different sequences with different lengths and vocabularies: translation, speech recognition, text-to-text transforms. Most general-purpose chat models skip the encoder and just concatenate everything into one decoder stream.
08 — Train vs. serve
One architecture, two very different runs
Training exploits parallelism. Generation is stubbornly sequential — which is why serving is a systems problem, not just a modeling one.
All positions at once
Feed a 4,000-token document. The causal mask lets every position predict its next token simultaneously, so one pass produces 3,999 training signals. Loss is cross-entropy against the true next token, averaged over positions, backpropagated through the whole stack.
No labels needed — the text is its own answer key. That's why the recipe scales: more text, more parameters, more compute, better model. The architecture has been stable for years; the scaling has not.
One token at a time
Generate, append, repeat. Press play to watch the loop.
Each square = one cached key/value pair.
The KV cache. Naively, generating token 500 would re-run the whole stack over all 500 tokens. The cache stores every earlier token's keys and values, so each new step computes only one new row — turning O(n²) work per step into O(n). The price is memory: the cache grows with context length and batch size, and for long contexts it can exceed the model weights. Techniques like GQA/MQA (share key/value heads across query heads) and MLA (compress them into a latent) exist mostly to shrink it.
09 — What actually changed since 2017
The skeleton survived. The parts didn't.
If you learn the 2017 diagram you can read a 2025 model — but you'd misread several boxes. Here's the drift.
| Component | 2017 original | Common now | Why |
|---|---|---|---|
| Normalization | LayerNorm, after residual | RMSNorm, before sublayer | More stable at depth; cheaper |
| Positions | Sinusoidal, added | RoPE, often with scaling | Relative positions generalize to longer contexts |
| Activation | ReLU | SwiGLU / GeGLU | Better quality per parameter |
| Attention | Full multi-head | GQA / MQA, sliding window, sparse | Shrink the KV cache, cut long-context cost |
| Biases | Yes, everywhere | Mostly removed | No measurable loss; fewer params |
| FFN | One dense MLP | Often MoE: route to a few of many experts | More parameters without proportional compute |
Mixture-of-Experts
The one structural departure worth knowing. Replace the single FFN with, say, 64 expert FFNs plus a small router that sends each token to the top 2. Total parameters balloon, but each token still touches only two experts, so compute per token stays flat. You get the knowledge capacity of a huge model at the speed of a small one — at the cost of memory and trickier training.
What didn't change
The block is still attention → add → FFN → add. Still next-token prediction. Still softmax over a vocabulary. Still residual streams carrying a single vector per token through every layer. Every paper that claims to replace the transformer either keeps this shape or quietly becomes one at scale.
10 — The bill
Why context is expensive
Attention compares every token with every other token, so its cost grows with the square of sequence length. The FFN grows linearly. Drag the slider to see which one dominates.
Forward-pass FLOPs per layer for the whole sequence — attention is 4n²d + 8nd², the FFN 16nd² — with dmodel = 4096 and a 4× expansion. Attention overtakes the FFN at n ≈ 2·dmodel, around 8k tokens here.
The quadratic wall
Doubling context quadruples attention compute and doubles KV-cache memory. Everything in modern inference engineering is an attack on this: FlashAttention recomputes attention in tiles that fit in SRAM so it never writes the full n×n matrix to memory; GQA shares key/value heads; sliding-window and sparse attention let each token see only a subset; linear attention and state-space hybrids like Mamba replace the n² term entirely.
Depth vs. width
More layers buy compositional depth (chains of reasoning); more width buys per-token capacity (facts, nuance). Empirically, depth is the better deal for reasoning and width for recall — which is why frontier models are both deep and wide and sparse. The FFN's d² cost also means a wider model is quadratically more expensive per token, while attention stays cheap at short context.
11 — Check yourself
Five questions, no grades
If these feel obvious, you've got the model. Each answer explains itself.
Want the full check — 34 questions, the arithmetic included, graded as you go? Take the comprehensive exam ↗