Why a '35GB' Quantized LLM Still OOMs an 80GB GPU: KV Cache, GQA vs. DeepSeek MLA, and the Real VRAM Math

Automated excerpt

How does a "35 GB" model run out of memory on an 80 GB GPU? Because model weights are only the static floor of LLM memory. KV Cache (64K context × batch 2 in FP16): 39. 1 GiB Activations + CUDA Runtime: ~4. 2 GiB Total VRAM Required: 83. 1 GiB ➔ Instant OOM on an 80GB A100/H100! (And if the model had used classic Multi-Head Attention with 64 KV heads instead of 8 GQA heads, that same KV cache would have required **312. 5 GiB! ) 4.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.