Memory is the toll booth
Prefill runs once, in parallel. Every token after that is its own round trip to memory to reload the model's weights and KV cache, which is why decode speed lives or dies on memory bandwidth.
An inference request has two phases that behave nothing alike. The prompt gets processed all at once, every token attending to every other token in parallel. That's prefill, and it happens exactly once per request. Then the model has to produce the reply, and that part is sequential: one token out, fed back in as input, one token out again. There's no parallelizing your way around it. Token 200 cannot start until token 199 exists.
The instinct is to think of both phases as a compute problem — more FLOPs, faster answer — because that's how training works. Decode doesn't work that way. Every single step has to reload the model's weights and the running history of prior activations, the KV cache, out of memory and onto the processor before it can produce one token. The arithmetic itself is cheap. The trip to memory, repeated once per output token, is what the clock is actually measuring.
decode is memory-bandwidth-bound, not FLOPs-bound — the bottleneck is the round trip, not the sum.
This is not abstract on the hardware I actually run. My agent setup lives on a Mac mini M4 with 16GB of unified memory, which is memory-constrained enough that it dictates the whole design. Weights get compressed from 16-bit floats down to 4-bit representations, and a router sends short, simple prompts to the small quantized model to keep tokens-per-second high and memory pressure low. The upgrade I've got planned — an M4 Pro with 48GB — is aimed straight at that ceiling: room to run larger, more complex models locally instead of routing everything through the smallest thing that fits.
16GB unified memory forces aggressive quantization; 48GB buys room for bigger models, not raw speed.
The same constraint shows up at enterprise scale, just wearing different clothes. On the team I work with at Google Cloud, integrating the most capable available models into products means the memory-bandwidth problem becomes a fleet problem: high- concurrency middleware, prompt engineering that trims what has to be reloaded per request, and token management systems, all aimed at the same two numbers a Mac mini router is also optimizing for — time-to-first-token and tokens-per-second — except now the third variable is cost, multiplied across every concurrent request.
Prefill is a sum you pay once. Decode is a toll you pay per token — and the toll booth is memory, not math.