How KV Cache Makes LLMs Fast, and Why It Eats GPU Memory

Automated excerpt

Same token, same position, same result, every time. The cache holds a key and a value for every token, in every layer, for every KV head. Attention needs the key and value of every earlier token, in every layer.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.