Why is the KV cache a thing?

The model has to read your whole prompt every time it picks a token. Why doesn't it choke? Because of a quiet trick almost nobody mentions in the docs.

The picture version

Six pictures for a reader who has never wondered how a chat reply streams. The prose below fills in the seams the pictures skip.

Why it exists

You’re forty messages deep in a chat. You paste in a long document and ask a question, there’s a pause, and then the answer streams out faster than you can read it. Notice how strange that is: the conversation above your question is thousands of tokens long, and the model is supposedly re-reading all of it to produce each word.

That’s the running example for this post — a 2,000-token prompt producing a 500-token answer — and here’s why it should bother you. An LLM takes your prompt and produces one next token. To produce the second token it looks at the prompt plus that first generated token. To produce the third, the prompt plus the first two. Every step extends the input by one token and runs the whole forward pass again.

Taken literally, your 500-token answer means 500 forward passes over sequences of length 2,001, 2,002, 2,003, … 2,500 — and each pass is, naively, quadratic in sequence length because of attention. That should be unusably slow. Instead tokens arrive at tens per second. How?

The answer is the KV cache. Every open inference stack you can read is built around it, and it rarely shows up in introductions. It’s why your tokens stream instead of stalling. It’s also why “context window” has a memory cost, why long prompts get expensive in a non-obvious way, and why the GPU serving you usually runs out of its own memory before it runs out of arithmetic.

Why it matters now

KV cache is load-bearing in a way most people don’t notice until something breaks:

• Throughput and latency in production. Without it, every generated token redoes the whole sequence, so the total work is a sum of quadratics that climbs fast as either the prompt or the answer gets longer. With it, each new token costs roughly one token’s worth of compute plus a read over the cache. The open serving stacks you can actually read — vLLM, TensorRT-LLM, llama.cpp — are all built around it.
• VRAM as the real bottleneck. People assume “context window” is a software limit. It’s partly an architecture and training choice, but at serving time the binding constraint is usually memory, and the KV cache is what’s occupying it. Doubling your context roughly doubles cache size per request, and a server holds one cache per concurrent request.
• Why prompt caching APIs exist. Providers ship “prompt caching” features that make a repeated prefix cheaper and faster. What they publish is the economics, not the implementation; what the open serving stacks do for the same effect is keep that prefix’s already-computed key/value state instead of recomputing it. Treat the internals of any hosted version as undocumented.
• Why speculative decoding works. Several proposed tokens can be verified in a single pass over the draft block against the cache, so checking a short draft costs far less than generating those tokens one at a time.

If you’re building anything that calls an LLM at scale, the cost model in your head should have “KV cache” in it.

The short answer

KV cache = a per-layer store of the key and value vectors for every token already in the sequence, reused so attention only has to compute K and V for the new token

Picture to keep: a long conference table where each seated person holds up a name card and a note. A new arrival reads every card and note before speaking — but nobody already seated ever rewrites theirs. Except that the cards aren’t free to hold up: they occupy the table, and the table is what runs out first.

A transformer’s attention layer computes three vectors per token: Q, K, and V. Q is what this token is asking about; K and V are what every token offers up to be attended to. The crucial asymmetry: when you generate token N+1, the K and V for tokens 1…N can’t change, because attention only looks backwards. They were already computed. The cache just keeps them around so you don’t redo the work.

How it works

Start from the naive loop and let each cost force the next move. Here’s transformer inference written out honestly:

prompt -> forward pass -> next token
prompt + token1 -> forward pass -> token2
prompt + token1 + token2 -> forward pass -> token3
...

Inside each forward pass, every attention layer does, for each input token, something like:

Q_i = x_i · W_Q
K_i = x_i · W_K
V_i = x_i · W_V
attention_i = softmax(Q_i · K_all / sqrt(d)) · V_all

Why the naive loop breaks: at step 500 you recompute K_i and V_i for all 2,499 earlier tokens, having computed exactly the same numbers 499 times already. Two things make that redundancy visible:

1. K_i and V_i depend on the hidden state at position i — which already folds in every earlier token — but they don’t depend on anything that comes after, because attention is causally masked. Appending token 2,501 cannot change them. So once computed for a given position, they’re frozen for the rest of generation.
2. To compute attention for the new token, you need the full K_all and V_all — the keys and values for every position so far. But you already computed almost all of them on previous steps.

Fix: keep K and V for every token, every layer, in GPU memory. On each new generation step:

• Run the forward pass on only the one new token.
• Compute its Q, K, V.
• Append the new K and V to the cache.
• Compute attention using the new Q against the full cached K and V.

That changes the per-step cost from “redo everything” to “do one token’s worth of compute, plus a single attention read against the cache.” The quadratic blow-up is gone for the generation phase.

There are two distinct phases people sometimes conflate:

• Prefill — the first forward pass over the whole prompt. This is expensive (roughly quadratic in prompt length) because nothing is cached yet. But it happens once.
• Decode — every subsequent step, generating one token at a time. This is the cheap, cache-using phase.

That split is why “time to first token” and “tokens per second after the first” are reported separately. Prefill cost lives in the first; cache amortization lives in the second.

What it costs

But you didn’t remove the work — you moved it into memory. That’s the trade the whole rest of modern inference is organized around. The cache size, per request, is roughly:

2 (K and V) × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element

Put your 2,500-token conversation through it with Llama 3.1 70B’s published config (80 layers, 8 KV heads, head dim 128, bf16): that’s 2 × 80 × 8 × 128 × 2 bytes ≈ 0.33 MB per token, so ~0.8 GB for this one chat. Push the same request to a 128k-token context and it’s ~42 GB. For scale: the weights alone are ~140 GB in bf16, so this model is already spread across at least two 80 GB H100 GPUs — and 140 GB of weights plus one maxed-out request’s 42 GB does not fit in that pair’s 160 GB. Now multiply the cache by however many requests you’re serving concurrently. This is why batching, paging (PagedAttention), quantization of the cache, and architectures like grouped-query attention exist. A large share of the engineering in modern inference engines is squeezing this one number.

Where it gets subtle

• It only works because attention is causal. Each token only attends to earlier tokens, so adding token N+1 doesn’t change anything about how tokens 1…N attended to each other. Bidirectional models (like classic BERT) can’t cache this way during inference because every token’s representation depends on the whole sequence in both directions.
• Cache correctness is fragile. Anything that changes the past — editing a previous token, inserting a system message after the fact, swapping model weights — invalidates the cache. This is one reason “edit your message” in a chat UI is implemented as starting a new generation, not patching mid-stream.
• Prompt caching across requests is the same trick at a different scope. If two requests share a long prefix (a system prompt, a long document), the key/value state for that prefix can be reused across them rather than recomputed — which is what open serving stacks do, and what makes a cache hit cheap. It’s the same asymmetry as within a request; what changed is how long the cached state has to survive.
• The exact numbers are deployment-specific, and often non-public. A model’s KV cache size per token depends on architecture details (KV head count after GQA, head dimension, dtype, whether the cache is quantized). Open-weights models publish those in a config file; hosted models generally don’t. Treat the size formula above as the shape of the cost, not a promise about any particular deployment.

You started with KV cache = a per-layer store of K and V for every token already in the sequence. What did the walk-through add? — + it only works because attention is causal, and + you didn't delete the work, you converted it into a memory bill. The first is the precondition nobody states; the second is why, for most serving setups, it’s GPU memory rather than raw arithmetic that decides how many conversations you can hold at once.

Check yourself

Before you go — two models are the same size, same layer count, same head dimension. One uses grouped-query attention with 8 KV heads; the other uses full multi-head attention with 64. Serving the same long conversations on one GPU, how does the number of concurrent users you can hold differ, and why?

And: you’re serving an API where every request begins with the same 8,000-token system prompt, then a short user question. Which phase does prompt caching help — prefill or decode — and roughly how much of the request does it eliminate?

Famous related terms

• Prefill vs. decode — prefill = one big pass over the prompt; decode = one-token passes using the cache. The two phases usually hit different limits (compute vs. memory bandwidth), which is why some inference engines schedule them separately.
• PagedAttention — PagedAttention ≈ virtual memory for the KV cache — chops the cache into fixed-size blocks so requests can grow without contiguous allocations. The core trick behind vLLM.
• Grouped-query attention (GQA) — GQA = multiple query heads sharing one KV head — directly shrinks KV cache size. Why most newer open models use it.
• Multi-query attention (MQA) — MQA = all query heads share a single KV head. The aggressive end of the same idea; trades quality for cache size.
• Speculative decoding — speculative decoding = small model proposes, big model verifies in parallel. Hits a sweet spot only because verification reuses the KV cache.
• Prompt caching — prompt caching = persist a prefix's KV cache between requests. The user-facing version of the same optimization.
• Memory bandwidth — decode speed ≈ bandwidth ÷ bytes read per token — the cache is the other thing your GPU reads on every token, alongside the weights.
• LLM — the thing whose attention layers this is all happening inside.
• Tokenization — defines the unit the cache is indexed by; longer tokens mean fewer cache entries for the same text.

Going deeper

• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023) — the primary source for “how much of the KV cache is actually wasted, and what does fixing it buy,” with measured fragmentation numbers.
• kipply, Transformer Inference Arithmetic — answers “can I derive the cache size and the decode cost myself, from the config file?” It works the arithmetic this post gestures at, in full.
• vLLM’s source (or llama.cpp, or TensorRT-LLM) — the rabbit hole: grep for kv_cache and follow the data structures to see what the cost formula above looks like as an allocator.