Why long context is expensive: the KV cache bill

Automated excerpt

Give it a 1M-token prompt and its KV cache alone needs 128 GB. Transformers re-read the whole conversation at every step, so each token gets stored as Key and Value vectors at every layer. Bigger models pay more: the 70B needs 320 KiB per token, 43 GB at 128K and 344 GB at 1M.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.