KV Cache: Why Your AI Chat Gets Slow
Have you ever wondered why your AI chat gets slower every turn? The KV cache is the model's memory of everything it has read, so nothing is read twice. By the end you will know why one changed word at the top throws that memory away, and how to order your prompt so it stays.
Have you ever wondered why your AI chat gets slower every turn? The KV cache is the model's memory of everything it has read, so nothing is read twice. By the end you will know why one changed word at the top throws that memory away, and how to order your prompt so it stays.
Does your chat really remember what you said?
Mia, Acme Shop's support chat, sounds like she remembers you, but her model keeps nothing between turns. The app resends the whole chat every time, and the model reads all of it at once (prefill). Then it writes the answer one token at a time (decode), and a token is a small word piece, as our LLM tokens post explains. Like re-reading a letter to add one sentence, every new token repeats attention, the look back at every earlier token. The Hugging Face Transformers docs on How caching works say the work per step grows quadratically (twice the text, four times the work) without a saved result, and linearly with one.
What is the KV cache, and what exactly does it save?
So what could Mia's model save to skip that redo? For every token it reads, it makes a key, which says what the token is about, and a value, which is what it hands over when a later token looks at it. The model does this in every layer, one of its stacked processing steps. The KV cache is those keys and values, stored so each step computes only the one new token, like a margin note kept for every token. That gives you four rules for Mia:
- Keep the notes: store the keys and values instead of recomputing them.
- Keep the start of the prompt identical, so the old notes still fit.
- Put the text that never changes first and the text that changes last.
- Add new turns at the end and never edit an earlier one.
Is it really that much faster, and is the answer the same?
Does the KV cache change anything? Let's test GPT-2 small, a 2019 open model, on Mia. Save the snippets below as one file, kv_cache_demo.py, run pip install onnxruntime==1.23.2 tokenizers==0.22.2 numpy==2.4.2, and keep GPT-2's two ONNX (portable model) files and tokenizer.json beside it. Mia's prompt is her seven rules (rules.txt) and the chat so far (chat.txt):
📄Text1234567You are Mia, the support assistant for Acme Shop, an online store that sells kitchen gear. Be polite and warm, and keep every answer to three sentences or fewer. Orders ship within 2 business days from our warehouse in Leeds, and tracking links are emailed on dispatch. Returns are free within 30 days of delivery if the item is unused and still in its original box. Refunds go back to the original payment method and take 5 to 7 business days to appear. Never promise a delivery date, only the shipping window, and never share another customer's details. If you do not know the answer, say so and offer to open a ticket for the support team.
📄Text123456Customer: Hi, my order 4417 has not arrived yet and I am getting worried. Mia: Sorry about the wait! Orders ship within 2 business days, and a tracking link is emailed when it leaves the warehouse. Did you get that email? Customer: Yes, but the tracking page has not changed since Monday. Mia: Thanks for checking. Parcels sometimes sit at a depot for a day or two, and I can open a ticket so the team can chase it for you. Customer: Please do. Also, can I still return the pan if it turns up damaged? Mia:
prefill reads every token and keeps the keys and values, step reads one new token and reuses them:
🐍Python12345678910111213141516import time import numpy as np import onnxruntime as ort from tokenizers import Tokenizer tok = Tokenizer.from_file("tokenizer.json") full = ort.InferenceSession("onnx/decoder_model.onnx") past = ort.InferenceSession("onnx/decoder_with_past_model.onnx") PAST = [i.name for i in past.get_inputs() if i.name.startswith("past")] rules, chat = open("rules.txt").read(), open("chat.txt").read() prompt = tok.encode(rules + chat).ids def prefill(ids): # read every token at once, keep keys and values out = full.run(None, {"input_ids": np.array([ids]), "attention_mask": np.ones((1, len(ids)), dtype=np.int64)}) ...
We write 40 tokens twice, with and without the cache:
🐍Python12345678910111213141516def write_without_cache(ids, n): # every new token re-reads the whole text ids, new = list(ids), [] for _ in range(n): new.append(int(prefill(ids)[0].argmax())) ids.append(new[-1]) return new def write_with_cache(ids, n): # read once, then one token per step logits, kv = prefill(ids) new = [] for _ in range(n): new.append(int(logits.argmax())) logits, kv = step(new[-1], kv) return new ...
📄Text12345prompt tokens: 272 no cache : 23.89s for 40 tokens with cache: 3.59s for 40 tokens same tokens: True cache: 24 arrays of shape (1, 12, 272, 64), 72 KiB per token
Same tokens, 6.7 times faster, because the notes were already written. The price is memory, 72 KiB per token here, and the Hugging Face Cache strategies docs call that growth the long-context bottleneck.
Why does one changed word at the top break everything?
That was one request, so what happens on turn 2? The vLLM Automatic Prefix Caching docs say a new request can reuse an earlier one's cache if it shares the same prefix, meaning the same opening text. Each entry was built from the tokens before it, so the cache is valid only up to the first token that differs, and every note after it goes stale. Most apps put a clock on line 1 of Mia's prompt:
📄Text12345Current time: 09:41 You are Mia, the support assistant for Acme Shop, an online store that sells kitchen gear. ... Customer: Please do. Also, can I still return the pan if it turns up damaged? Mia:
The fix moves that one line down, above the customer's new message:
📄Text123456You are Mia, the support assistant for Acme Shop, an online store that sells kitchen gear. ... Mia: Thanks for checking. Parcels sometimes sit at a depot for a day or two, and I can open a ticket so the team can chase it for you. Current time: 09:41 Customer: Please do. Also, can I still return the pan if it turns up damaged? Mia:
This code cuts the cache back to the first different token and computes the rest:
🐍Python12345678910111213141516def first_difference(old, new): # the cache is valid only up to here n = 0 while n < min(len(old), len(new)) and old[n] == new[n]: n += 1 return n def reuse_and_finish(old_ids, old_kv, new_ids): n = first_difference(old_ids, new_ids) kv = [a[:, :, :n] for a in old_kv] # cut the cache back to the first different token for token in new_ids[n:]: # compute only what is left, one token per call logits, kv = step(token, kv) return n, logits lines = chat.splitlines(keepends=True) def with_clock(minute, where): ...
📄Text12clock at bottom: prompt 279 tokens, reused 253, computed 26, same logits: 4.6e-05 clock at top: prompt 279 tokens, reused 5, computed 274, same logits: 9.2e-05
At the bottom Mia reuses 253 of 279 tokens, at the top 5. Her logits (raw next-token scores) match to 1e-4: same answer, less work.
When is the cache the wrong tool?
So should Mia cache everything? No. The vLLM Automatic Prefix Caching docs say prefix caching reuses a prompt's saved start. That speeds up prefill, which is reading the prompt, but not decode, which is writing the answer. It does nothing for a short or one-off prompt or one whose top changes every call. The Anthropic Prompt caching docs add that, as of November 2026, entries expire after at least 5 minutes idle, and prompts below a minimum length are not cached.
Tomorrow, move every per-request line in your chat's system prompt below the stable text and send two turns. In the second reply's Anthropic usage fields, cache_read_input_tokens above 0 means the cache was used, cache_creation_input_tokens counts tokens just stored, and input_tokens counts the rest.
Conclusion
So why did Mia get slower every turn, and did she ever remember your chat? No: her model re-read the whole text, and the KV cache, its saved keys and values, lets it read only the new tail. One changed word at the top, like her clock, ruins every note, so keep stable text first, then trim what is left with our Context Optimization post. Next time: quantization, shrinking a model to fit a GPU.
Sources
- Hugging Face Transformers docs, How caching works
- Hugging Face Transformers docs, Cache strategies
- vLLM docs, Automatic Prefix Caching
- Anthropic docs, Prompt caching
Enjoyed this post?
Get the next one in your inbox. No spam, unsubscribe any time.
Comments
0 comments
