Interactive · no math degree required

How a transformer actually works

A stack of identical blocks. One operation that lets every word look at every other word. Nothing else. Eleven sections, nine things to play with — and you'll be able to read a model diagram without squinting.

scroll

01 — The one idea

Everything is communication, then computation

In 2017, Attention Is All You Need replaced recurrence with a deceptively simple mechanism: let every token decide, at runtime, which other tokens matter to it.

Before: one word at a time

An RNN reads left to right, folding each new word into a running hidden state. Word 400 must wait for words 1–399, and the early ones get squeezed through 400 multiplications. Long-range dependencies fade; training can't parallelize over the sequence.

After: everything at once

Attention compares every token with every other token directly — one hop, regardless of distance. No sequential dependency, so the whole sequence processes in parallel on a GPU. That single change is why models could suddenly get enormous.

◆

The whole architecture in one sentence: alternate two operations — attention, where tokens exchange information, and a feed-forward network, where each token thinks about what it just received — and repeat that block 12 to 100+ times.

One thing to hold onto: the architecture has barely changed since 2017. Almost every "new" model is the same block with better normalization, better position encoding, and better attention kernels. Learn the block and you've learned all of them.

02 — The pipeline

Follow one sentence through the model

From raw characters to a probability for the next token. Click any stage.

text → tokens → vectors → [ block × N ] → logits → next token

Notice there is no "understanding" step and no grammar module. Everything above is learned matrix multiplication, and the only structural decisions are how wide, how deep, and how many tokens can look at each other.

03 — Tokens & positions

The model never sees letters

Text is chopped into subword units from a fixed vocabulary, each mapped to an integer. Type anything below.

Why subwords?

A character vocabulary makes sequences painfully long; a whole-word vocabulary can't handle typos, names, or "unbelievably". Subword tokenizers (BPE, SentencePiece) land in between: common words stay one token, rare ones split into reusable pieces. unbelievable → un + believ + able.

Those pieces are shared, so the model learns morphology for free: -ing, -ed, un- behave consistently wherever they appear. Typical vocabularies run 32k–200k entries.

Why positions?

Attention is permutation-invariant: it computes dot products, and dot products don't care about order. Feed it "dog bites man" and "man bites dog" shuffled and you get the same multiset of outputs. Position information must be injected explicitly.

Old way: add a sinusoidal or learned vector to each embedding. Modern way: RoPE, which rotates each query and key by an angle proportional to its position — so the attention score between two tokens depends on their relative offset, which generalizes better to lengths never seen in training.

04 — Self-attention

The only place tokens talk

Each token asks a question (query), every token advertises what it has (key), and the ones that answer well contribute their contents (value).

Q
Query — “what am I looking for?”

One learned projection of the token's vector. It is compared against every key.

K
Key — “what do I offer?”

Another projection, one per token. The query·key dot product measures relevance.

V
Value — “what do I pass on?”

A third projection holding the actual content. Values are averaged using the attention weights.

Attention(Q, K, V) = softmax( QKT / √dk ) · V
√

Why divide by √dk? Dot products of random vectors grow with dimension: at d=64, scores land around ±8 instead of ±1. Softmax of large numbers is nearly one-hot, so almost all gradient vanishes and learning stalls. Scaling keeps scores in a sane range. It's a tiny detail that decides whether training works.

INTERACTIVE

Who is each word looking at?

Click a word to make it the query · hover bars to trace

What attention is not

It isn't a lookup table of grammar rules, and the weights aren't "explanations". They're a soft, learned routing: token A copies a weighted blend of token B's value, and the next layer does something useful with that blend. Meaning emerges across layers, not in any single head.

The party analogy

Every token walks into a room and asks "who here knows about what I need?" It scores everyone's one-line self-description, spends its attention budget on the best matches, and comes away with a summary of what they said. Then it thinks privately (feed-forward), and the next room has a new question.

05 — Multi-head attention

Many conversations at once

One attention operation can only express one notion of relevance per token. So run several in parallel at lower dimension, then concatenate. Same cost, far more expressive.

headi = Attention(QWiQ, KWiK, VWiV)·MHA = Concat(head1 … headh)WO

Same sentence, three heads. Each panel is a full attention matrix — rows are queries, columns are keys, brightness is weight. Hover a cell.

Real models have 16–128 heads, and nobody assigns them jobs. Training discovers specializations on its own: previous-token heads, induction heads that copy patterns, heads that track quotation and syntax, and "attention sink" heads that dump weight on the first token as a no-op. A 32-head layer is 32 partly-overlapping views of the same sequence, and the output projection learns how to combine them.

◈

Why split instead of one big head? One head of width 4096 has one softmax distribution — it must commit to a single relevance pattern. Thirty-two heads of width 128 each get their own distribution, and together they can track syntax and coreference and position simultaneously. The total parameter count and FLOPs are identical.

06 — The rest of the block

Attention is only half of it

Three more ingredients turn the idea into a trainable deep stack: normalization, residual connections, and a private thinking step.

one transformer block
→
× N layers — every layer keeps the same shape
Click any block to learn what it does. Both sublayers sit inside one block; the stack repeats it N times.

Residual connections

Every sublayer output is added back to its input: y = x + f(x). Gradients flow straight down that addition, so a 96-layer network trains about as easily as a 6-layer one. This single line, borrowed from ResNet, is what made depth practical.

Normalization

LayerNorm (or RMSNorm) re-scales each token's vector so activations don't drift as depth grows. RMSNorm drops mean-centering — cheaper, equally good — and is what most current models use.

The feed-forward network

After attention mixes information across positions, the FFN transforms each position independently: expand to ~4× width, apply a nonlinearity, project back. It holds roughly two-thirds of all parameters.

Intuition: attention finds the relevant context; the FFN computes with it. Mechanistic interpretability suggests the first matrix detects features, the activation selects which are present, and the second writes associated facts — closer to a key-value memory than to a classic MLP.

07 — Three shapes

Same block, different masks

The mask is the only real architectural fork. It decides which tokens are allowed to see which other tokens — and therefore what the model can be trained to do.

Encoder-onlyBERT

Sees everything

Bidirectional attention. Trained by hiding ~15% of tokens and predicting them. Produces rich contextual representations — excellent for classification, retrieval, and embeddings. Can't generate fluently.

when to reach for it

Any task where you need a good vector for a span of text: search, reranking, NER, sentiment, clustering. Smaller, cheaper, and often more accurate than asking a chat model.

Decoder-onlyGPT · Llama · DeepSeek

Sees only the past

A causal mask blocks every future position. The payoff: one forward pass yields a next-token prediction at every position, so one sequence gives N training signals. This is why it won.

why generation works

Because each position was trained to predict only from its own past, the model can be run on a prefix and asked for one new token — then that token is appended and the process repeats. Same function, applied incrementally.

Encoder–decoderT5 · Whisper

Reads, then writes

An encoder reads the input bidirectionally; a decoder generates causally and periodically attends into the encoder's output through cross-attention.

when it still wins

When input and output are genuinely different sequences with different lengths and vocabularies: translation, speech recognition, text-to-text transforms. Most general-purpose chat models skip the encoder and just concatenate everything into one decoder stream.

08 — Train vs. serve

One architecture, two very different runs

Training exploits parallelism. Generation is stubbornly sequential — which is why serving is a systems problem, not just a modeling one.

TRAINING

All positions at once

Feed a 4,000-token document. The causal mask lets every position predict its next token simultaneously, so one pass produces 3,999 training signals. Loss is cross-entropy against the true next token, averaged over positions, backpropagated through the whole stack.

No labels needed — the text is its own answer key. That's why the recipe scales: more text, more parameters, more compute, better model. The architecture has been stable for years; the scaling has not.

INFERENCE

One token at a time

Generate, append, repeat. Press play to watch the loop.

Each square = one cached key/value pair.

⚡

The KV cache. Naively, generating token 500 would re-run the whole stack over all 500 tokens. The cache stores every earlier token's keys and values, so each new step computes only one new row — turning O(n²) work per step into O(n). The price is memory: the cache grows with context length and batch size, and for long contexts it can exceed the model weights. Techniques like GQA/MQA (share key/value heads across query heads) and MLA (compress them into a latent) exist mostly to shrink it.

09 — What actually changed since 2017

The skeleton survived. The parts didn't.

If you learn the 2017 diagram you can read a 2025 model — but you'd misread several boxes. Here's the drift.

Component2017 originalCommon nowWhy
NormalizationLayerNorm, after residualRMSNorm, before sublayerMore stable at depth; cheaper
PositionsSinusoidal, addedRoPE, often with scalingRelative positions generalize to longer contexts
ActivationReLUSwiGLU / GeGLUBetter quality per parameter
AttentionFull multi-headGQA / MQA, sliding window, sparseShrink the KV cache, cut long-context cost
BiasesYes, everywhereMostly removedNo measurable loss; fewer params
FFNOne dense MLPOften MoE: route to a few of many expertsMore parameters without proportional compute

Mixture-of-Experts

The one structural departure worth knowing. Replace the single FFN with, say, 64 expert FFNs plus a small router that sends each token to the top 2. Total parameters balloon, but each token still touches only two experts, so compute per token stays flat. You get the knowledge capacity of a huge model at the speed of a small one — at the cost of memory and trickier training.

What didn't change

The block is still attention → add → FFN → add. Still next-token prediction. Still softmax over a vocabulary. Still residual streams carrying a single vector per token through every layer. Every paper that claims to replace the transformer either keeps this shape or quietly becomes one at scale.

10 — The bill

Why context is expensive

Attention compares every token with every other token, so its cost grows with the square of sequence length. The FFN grows linearly. Drag the slider to see which one dominates.

Forward-pass FLOPs per layer for the whole sequence — attention is 4n²d + 8nd², the FFN 16nd² — with dmodel = 4096 and a 4× expansion. Attention overtakes the FFN at n ≈ 2·dmodel, around 8k tokens here.

attention
—
feed-forward
—
verdict
—

The quadratic wall

Doubling context quadruples attention compute and doubles KV-cache memory. Everything in modern inference engineering is an attack on this: FlashAttention recomputes attention in tiles that fit in SRAM so it never writes the full n×n matrix to memory; GQA shares key/value heads; sliding-window and sparse attention let each token see only a subset; linear attention and state-space hybrids like Mamba replace the n² term entirely.

Depth vs. width

More layers buy compositional depth (chains of reasoning); more width buys per-token capacity (facts, nuance). Empirically, depth is the better deal for reasoning and width for recall — which is why frontier models are both deep and wide and sparse. The FFN's d² cost also means a wider model is quadratically more expensive per token, while attention stays cheap at short context.

11 — Check yourself

Five questions, no grades

If these feel obvious, you've got the model. Each answer explains itself.

0 / 5 correct

Want the full check — 34 questions, the arithmetic included, graded as you go? Take the comprehensive exam ↗