How do transformers actually work?
Every word a model writes is one full trip through the same stack of layers that every transformer shares. Follow a single token from text to prediction to see how a transformer works, then why this one architecture still beats the alternatives at scale.
A model does not know its answer in advance. It rebuilds a probability for the next token from scratch on every pass, which is why generation is sequential and why each token carries a hard compute and memory cost.
One word at a time, one pass at a time
A forward pass is a single trip of data through the model, from input to a prediction of the next token. The word forward is in contrast to the backward pass, which happens only during training to update the weights. At inference the weights are frozen, so the model only ever runs forward passes.
The fact to hold onto: every token a model generates is one entire forward pass through every layer. A 500-word answer is hundreds of passes, each one rebuilding the next-word prediction from the whole context so far. Picture a fixed assembly line whose stations never change; only the token moving down it does.
The new token is appended to the prompt, and the entire pass runs again. Every word you read is one more trip through the whole stack.
The model has no draft of the sentence. It commits to one token, then reconsiders everything.
At inference the parameters are read-only. Only the token moving through them changes.
A long answer is hundreds of passes, which is why generation is sequential and priced by the token.
First, words become vectors
The model does not read text. A tokenizer splits the input into sub-word pieces, each mapped to an integer. The model holds one giant lookup table, the embedding matrix, with a row for every entry in its vocabulary. A token id simply fetches its row: a list of a few thousand numbers that places the token in a space where meaning is distance.
From here on there are no words, only vectors and matrix multiplications. This first step is a pure memory fetch, not arithmetic, one row pulled from memory per token.
No arithmetic yet. The id just indexes a row, one lookup from memory per token.
Similar words land near each other in this space, so math on vectors can reason about meaning.
A running vector flows down a stack of identical layers
Each token vector rides down the stack as a residual stream. Every layer reads the current vector, computes an update, and adds it back, so layers contribute edits rather than overwriting. By the end, that vector has accumulated everything the model worked out about the token in context.
Every layer runs the same two-part move, dozens to a hundred times over: first attention, then a feed-forward network. The repetition is deliberate. A small, repeated set of operations is exactly what lets hardware and compilers optimize the hot paths relentlessly.
Each block computes an edit and adds it to the vector, so information accumulates instead of being replaced.
Every layer is identical in structure. The repetition is what makes the hardware easy to optimize.
Early layers see raw words; later layers see who did what to whom, built on the layers below.
Attention: each token decides who to listen to
Attention is the only step where tokens exchange information. From each token vector, three weight matrices produce three new vectors: a Query (what am I looking for), a Key (what do I offer), and a Value (what I pass along if chosen). It works like a search. A token compares its Query against every other token Key to score relevance, softmax turns those scores into weights that sum to one, and the token takes a weighted sum of everyone Values. This is how the word it can reach back and pull in the meaning of cat from three sentences earlier.
This step is also where the key-value cache is born. The Keys and Values of past tokens are needed again for every future token, so the model stores them rather than recomputing. That stored pile grows with context length and is the single biggest variable cost in serving a model.
The output is a weighted blend of every token’s Value, dominated by cat. This is how “it” picks up that it refers to the cat.
Three matrices turn each token into a search request, a label, and the content it passes on if chosen.
Everywhere else each token is handled alone. Attention is where tokens actually exchange information.
Storing every earlier token's Key and Value, so they aren't recomputed, is exactly the KV cache.
The feed-forward network is where the token thinks
After attention has mixed in context, each token passes alone through a feed-forward network: two matrix multiplications with a nonlinearity between them, projecting the vector up into a wider space and back down. If attention gathers the relevant information, the feed-forward network processes it. This is where most of the model parameters, and most of its stored knowledge, actually live.
In a mixture-of-experts model this stage splits into many expert networks, of which only a few fire per token, which is how a model can hold far more parameters than it pays to run on any single token. The result is added back to the residual stream, completing one layer, and the whole attention-then-network move repeats for the next layer.
Most of the model's parameters sit in these two matrices. This is the model applying what it learned.
Unlike attention, this step does not mix tokens. It processes each one independently.
In a mixture-of-experts model this stage is many feed-forward networks, of which only a few fire per token.
The last vector becomes a probability
After the final layer, the token vector is multiplied by an unembedding matrix, roughly the reverse of the first step, producing one raw score, a logit, for every word in the vocabulary. Softmax turns those logits into a probability distribution over every possible next token.
The model then samples one token from that distribution, with a temperature setting that controls how safe or adventurous the pick is. That chosen token is the output, and it is fed back in so the next pass can predict the word after it. The model never plans the whole sentence; it commits one token at a time.
The sampler draws one token, here mat. A temperature dial decides how often it strays from the top choice. Then the token is fed back in for the next pass.
Raw scores over the whole vocabulary become a clean probability distribution that sums to one.
Low temperature picks the safe top word; high temperature makes the model more adventurous.
Even a 61% favorite is not certain. Sampling is why the same prompt can yield different answers.
The same pass runs at two very different speeds
The same machinery has two regimes. Prefill digests the entire prompt in one pass, so all prompt tokens move through together as large matrix-by-matrix multiplications that saturate the chip. It is compute-bound. Decode then generates the answer one token at a time, reusing the cached Keys and Values, so each pass is a skinny matrix-by-vector operation with little arithmetic per byte of weights it must stream.
That makes decode memory-bandwidth-bound: the limit is how fast the chip can pull weights and cache through the compute, not how many multiplications it can do. The asymmetry is why modern systems split prefill and decode onto different hardware, and why nearly every architecture advance that shrinks the cache or the bytes per weight is really an attack on this one bottleneck.
Many tokens processed together as a big matrix by matrix multiply. The chip is busy doing arithmetic.
One skinny token reuses the cache, so little arithmetic per byte. The limit is how fast weights stream from memory.
The two phases stress different resources, so modern systems run them on different hardware pools.
Nearly every architecture trick that shrinks the cache or the bytes per weight is an attack on this one bottleneck.
Why this architecture won: attention made context parallel
Before transformers, sequence models processed tokens more sequentially. Attention let tokens compare themselves to other tokens in the context at the same time.
That parallelism was the breakthrough that mattered for scaling. It matched the hardware direction of the industry.
The architecture is simple enough to optimize
Transformer blocks repeat a small set of operations: attention, feed-forward networks, residual connections, normalization, and projections. That repetition lets hardware and software teams optimize the hot paths relentlessly.
A clean architecture compounds with compilers, kernels, memory layouts, and accelerator design.
The weakness is memory
Attention has to manage relationships across context, and inference has to carry key-value cache state forward. Long context therefore creates memory pressure even when compute looks available.
That is why so many transformer improvements are really memory improvements wearing model-research clothes.
Alternatives have to win the system, not the paper
State-space models, recurrent designs, diffusion-language hybrids, and other architectures can improve specific trade-offs. The frontier bar is harder: quality, trainability, serving economics, tooling, and hardware fit all at once.
A replacement does not merely need a clever mechanism. It needs a full stack that labs can trust at scale.
The likely future is hybrid
The transformer may not disappear. It may become one component in a broader model system, joined by retrieval, tools, memory, verifiers, specialized modules, and agent scaffolds.
The point is not that transformers are eternal. It is that they remain the best default until another architecture beats the whole operating system around them.
The hybrid is already shipping
The hybrid future is not a forecast, it is in production. Nvidia's Nemotron 3, along with recent models from Qwen and Kimi, combines transformer attention with state-space (Mamba-style) layers, and the two do genuinely different jobs. A state-space layer compresses the whole sequence into a fixed-size running summary. That is cheap, and the very constraint of a fixed-size summary forces the model to learn what matters globally, so state-space layers are good at impressionistic, gist-of-the-whole-thing understanding. Attention keeps every token individually available and can reach back to pull out one exact detail with no lossy compression.
Nvidia's team reports the best mix is mostly state-space with a little attention, and that the combination is not merely faster but measurably smarter than either mechanism alone. That is the useful correction to the standard mental model: attention is not the whole of a transformer, it is one operation (precise, expensive retrieval) increasingly paired with cheaper operations that handle the parts of the job attention overpays for. The winning architecture is a division of labour, not a single mechanism, and the industry has already started converging on it.
Source: Bryan Catanzaro (VP, Applied Deep Learning Research, Nvidia), The MAD Podcast with Matt Turck, July 2, 2026; Nvidia hybrid Mamba-transformer research (2024).